Two things:
You're right, he doesn't fit Payton's system, which is a modified West Coast. Right now, the Broncos are 5-3 and while Nix is not playing lights out, he's doing what Payton wants him to do. And he's going to be much better next year.
Payton demands a certain type of locker room. Wilson had gotten out of control in Denver and even after Payton thumped him, the culture had to be changed. I think Pittsburgh's offense fits Wilson better and I think Payton moving on from him, may have remotivated him to refocus on football.